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ABSTRACT 

The  Air  Force  Research  Laboratory  is  conducting  research  and  development  of  a  computer-based  simulation 
capability  to  support  training  in  decision-making  and  team  coordination  for  security  forces  ground  operations. 
Simulation  software  supports  the  interaction  (over  a  local  area  network)  of  trainees  with  each  other  and  with 
computer-generated  forces  (CGF’s)  that  simulate  behavior  and  communications  of  enemy,  neutral,  and  friendly 
troops  and  civilians.  Radio  functions  allow  multi-channel  communication  among  instructors,  trainees,  and  CGF’s. 
Trainees  can  practice  decision-making  and  team  coordination  in  a  number  of  scenarios  with  varying  threat  and 
environmental  conditions.  Current  systems  are  too  costly  for  training  large  numbers  of  security  forces  because  they 
require  an  on-site  technician  to  develop  simulation  exercises,  control  the  exercise,  serve  as  role  players,  task  CGF’s, 
and  support  after  action  reviews.  Consequently,  design  and  development  of  a  simulation  control  interface  that  can 
be  directly  used  by  instructors  and  trainees  is  an  important  R&D  objective.  For  the  security  forces  simulation 
capability  the  goal  is  to  design  a  control  interface  that  instructors  can  learn  to  use  in  two  hours  and  trainees  can  learn 
to  use  in  thirty  minutes.  To  achieve  this  goal,  a  Windows-based  control  interface  (with  a  number  of  video  game 
features)  was  adopted  as  the  initial  point  of  departure.  Menu  options  were  developed  to  correspond  to  the  standard 
mission  planning  procedures  used  by  security  forces  and  drag-and-drop  functions  were  developed  to  replace  menu 
options  to  contribute  to  usability.  We  have  conducted  an  evaluation  of  the  initial  user  interface  with  school 
instructors  and  trainees.  The  paper  describes  the  emerging  control  interface,  the  approach  to  and  outcomes  from  a 
field  evaluation  of  the  interface  to  include  actual  times  required  for  instructors  and  trainees  to  learn  to  use  the 
system  and  instructors’  acceptability  evaluations.  Lessons  learned  shed  light  on  critical  human-machine  interface 
design  issues  for  computer-based  real-time  training  simulations. 
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INTRODUCTION 

United  States  Air  Force  security  forces  are  responsible 
for  military  police  services,  installation  security, 
airbase  defense,  military  working  dog  functions,  and 
combat  arms  training  and  maintenance.  They  have  a 
central  role  in  force  protection.  Whether  security 
forces  discover  an  improvised  explosive  device  in  a  car 
or  confront  representatives  of  a  non-governmental 
organization  at  an  entry  control  point,  such  situations 
are  reported  to  a  command  post  whether  it  is  called 
central  security  control,  base  defense  operations  center, 
or  a  law  enforcement  desk.  Once  situations  are 
reported,  quick  and  accurate  decisions  by  security 
forces  leaders  and  decision-makers  are  critical  to 
handling  the  situation  properly  and  fundamental  to  the 
protection  of  personnel  and  assets. 

Command  post  exercises  are  periodically  conducted  to 
train  personnel  to  respond  to  diverse  situations. 
However,  training  decision-makers  requires  a  large 
number  of  subordinate  personnel  to  be  deployed  over  a 
large  area.  Consequently,  the  command  and  control  of 
security  forces  continues  to  be  recognized  as  a  high- 
emphasis  training  area  (Weeks,  Garza,  Archuleta,  and 
McDonald,  2001).  As  a  result  of  established  training 
needs  and  technology  opportunities,  the  Air  Force 
Research  Laboratory  is  conducting  research  and 
development  of  an  interactive  simulation  capability 
(Weeks  and  McDonald,  2002).  Computer-driven, 
simulations  are  regarded  as  an  affordable  way  to 
accelerate  acquisition  of  expertise  in  leadership, 
decision-making,  and  team  coordination;  but,  fielding 
such  a  capability  depends  on  its  usability  and  proven 
training  effectiveness. 

Guiding  Principles 

Since  security  forces  are  the  second  largest  career  field 
in  the  Air  Force  and  there  are  limited  funds  for  their 
training,  affordability  is  a  major  design  constraint.  To 
meet  these  affordability  goals,  trainees  and  instructors 
should  be  able  to  operate  the  simulation  without  an  on¬ 
site  technician.  Consequently,  usability  is  a  critical 


aspect  of  this  development  effort.  After  extensive 
interviews  of  security  forces  personnel,  we  have 
developed  the  following  guiding  principles  for  a 
computer  based  simulation  for  training  Air  Force 
Security  Forces  leaders  and  decision-makers: 

•  Learning  how  to  use  the  simulation  must  require  a 
negligible  amount  of  time  relative  to  the  time  spent 
training 

•  Simulation  interfaces  must  be  simple  and  intuitive 

•  Show  trainees  only  what  they  would  see  in  the 
field;  no  omniscient  views  of  threat  locations 

•  The  user  interface  must  simulate  the  primary  tools 
used  in  the  field,  radio  communications  and  a 
tactical  map 

Training  Simulation 

In  order  to  understand  the  usability  challenge,  a 
description  of  the  training  simulation  is  presented 
below.  An  illustration  of  the  training  device  is 
presented  at  Figure  1 .  It  consists  of  standard  personal 
computers  connected  by  a  local  area  network.  Trainees 
are  illustrated  as  a  shift  leader  and  subordinate  flight 
leaders  but  could  alternatively  be  a  law  enforcement 
desk  sergeant  and  field  officers,  a  flight  leader  and 
subordinate  squad  leaders,  or  a  defense  force 
commander,  operations  officer,  and  area  supervisors. 
The  capability  is  being  developed  as  a  multi-echelon, 
command  and  control,  training  device  to  support 
different  training  requirements. 

The  computer  display  simulates  the  tactical  map  used 
in  the  field.  Trainees  read  an  Operations  Order  and 
place  overlay  symbols  on  the  map  using  the  mouse  and 
menu  selections.  Placing  these  symbols  on  the 
simulated  map  populates  the  virtual  world  with  assets 
(e.g.  riflemen,  fighting  positions,  barriers,  sensors, 
entry  control  points)  in  their  areas  of  responsibility. 
They  also  use  menu  commands  to  initiate  actions  such 
as  patrols  and  to  set  rules  of  engagement  for  the 
computer  generated  forces  (CGF’s)  under  their 
command.  Once  the  trainees  have  completed  their 
plans,  the  instructor  initiates  the  exercise  in  which 
threat,  friendly  and  neutral  CGF’s  enter  the  facility. 
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Figure  1.  The  Training  Device 


Friendly  CGF’s  under  the  command  of  the  trainees 
issue  verbal  situation  reports  (SITREPS)  over  their 
assigned  radio  channels.  Trainees  hear  the  SITREPS 
over  their  headsets,  see  sensor  alarms  and 
communicate  with  other  trainees  using  the  simulated 
radio  channels.  They  then  make  decisions  and  issue 
orders  to  their  CGF’s  via  menu  selections. 

Subject  matter  experts  indicated  that  one  of  the  most 
difficult  tasks  for  young  leaders  and  decision-makers  to 
learn  is  situational  awareness  (the  ability  to  mentally 
picture  the  locations  and  movements  of  threats, 
neutrals  and  friendlies  based  on  SITREPS  and  sensor 
alarms.  During  the  planning  phase,  trainees  place 
friendly  icons  on  the  simulated  map  just  as  they  would 
an  actual  map. 

When  the  exercise  starts,  these  icons  do  not  move 
when  commanded  to  do  so  because  annotations  on  a 
real  tactical  map  do  not  move  automatically.  Trainees 
move  the  friendly  icons  based  on  mental  dead¬ 
reckoning  and  location  reports.  Needless  to  say,  the 
actual  locations  of  threats  and  neutrals  are  not 
displayed  on  the  simulated  tactical  map,  because 
trainees  would  not  learn  situational  awareness.  As 


friendly  CGF’s  issue  SITREPS,  trainees  can  place 
suspected  location  icons  for  threats  and  neutrals  on  the 
simulated  map,  just  as  they  do  in  the  real  world. 
During  the  AAR,  all  verbal  communications  over 
simulated  radio  channels  are  played  back  in  real  time. 
In  addition,  the  actual  locations  of  threats,  friendlies 
and  neutrals  are  replayed  in  real  time  along  with  the 
suspected  location  icons.  These  comparisons  are 
believed  to  promote  the  development  of  situational 
awareness  skills. 

The  simulation  has  over  450  functions  to  be  controlled 
by  the  trainees  and  instructor  during  the  mission 
planning,  mission  execution  and  AAR.  This  is  the 
reason  usability  has  been  given  such  a  high  priority  in 
this  development  effort. 

Usability  Issues 

Since  there  is  no  rigid  sequence  required  for 
conducting  mission  planning,  we  chose  to  use  a  mixed 
initiative  dialogue  with  direct  manipulation  of  data. 
We  chose  menu  selections  as  opposed  to  a  user 
command  language  because  of  our  desire  to  have 
trainees  learn  to  use  the  simulation  quickly  (Smith  & 
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Aucella,  1983;  Ambardar,  1991)  and  begin  training. 
We  chose  menu  selections  to  control  CGF’s  because 
English  language  dialogue  is  “unsuitable  where  an 
operator  input  must  be  interpreted  with  precision 
(because  of  ambiguity)”  (Boff  &  Lincoln,  1988). 
Although  speech  understanding  systems  have  come  a 


long  way  in  recent  years,  this  technology  is  still  an 
active  research  topic  and  not  ready  for  use  as  a  routine 
dialog  tool  in  a  command  and  control  simulation.  The 
menu  hierarchy  was  designed  to  be  wide  and  flat  with 
a  maximum  of  10-15  options  at  each  level  per  the  DoD 
Technical  Architecture  Framework  (1996). 


I  3  □  Mm'tth _ |  ■  i [M  n.tAHE^fei  ■  ■  ■  1 1  3  C  '■  >*  a',  ■  ■  C 

Figure  2.  The  Simulation  Control  Interface 


RESEARCH  OBJECTIVES 

Research  objectives  include  development  of  a  usable 
control  interface,  realistic  behaviors  for  CGF’s, 
development  of  simulation  exercises  to  support 
learning  objectives,  and  evaluation  of  system  usability, 
model  validity,  and  training  effectiveness.  The 
research  and  development  project  includes  multiple 
field  evaluations  with  participation  of  end  users.  The 
development  strategy  is  to  collect  evaluation  feedback 
and  apply  it  to  refinement  of  the  capability  in  an  effort 
to  accelerate  transition  to  the  field. 

The  initial  evaluation  (and  the  issue  addressed  in  this 
paper)  consists  of  assessments  of  the  usability  of  the 
simulation  control  interface.  Usability  assessments 
include  measures  of  training  time  for  instructors  and 
trainees  and  instructors’  ratings  of  the  usability  of  the 


control  interface.  The  evaluation  concludes  with 
assessments  of  the  overall  value  of  the  training  device 
and  draft  training  scenarios  for  mission  planning  and 
execution  of  the  defense.  The  purpose  of  this  paper  is 
to  describe  outcomes  obtained  from  initial  field 
evaluations.  Summary  conclusions  are  expected  to 
shed  light  on  lessons  learned  and  the  promise  of 
computer-based  training  simulations  for  the  future. 

USABILITY  EVALUATION  APPROACH 

Brewer,  Armstrong,  and  Steinberg  (2002)  state 
“usability  testing  ...  verifies  that  a  system  or  product 
design  meets  our  expectations  for  usefulness  and 
satisfaction  before  we  move  into  production  (p  403).” 
They  define  usability  as  “the  degree  to  which  the 
design  of  a  device  or  system  may  be  used  effectively 
and  efficiently  by  a  human  (p  403)”  and  point  out  that 
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the  important  issue  in  arriving  at  a  definition  is  how  to 
measure  usability  so  that  measurements  can  be  used  to 
improve  the  design.  Brewer,  et  al.  (2002)  outline  three 
general  approaches  to  usability  evaluation  including 
surveys  (using  self-report  data  collection  methods), 
usability  inspections  (specialists  scrutinize  a  design 
according  to  a  systematic  approach  and  judge  its 
acceptability  against  certain  criteria),  and  experimental 
tests  (based  on  quantifying  operator  performance  using 
controlled  data  collection  techniques). 

Usability  inspection  best  describes  the  approach  to  the 
initial  evaluation.  The  specialists  were  instructors 
assigned  to  different  security  forces  training  squadrons. 
They  were  first  trained  to  use  the  simulation  control 
interface;  then,  they  evaluated  the  interface  on  critical 
criteria,  identified  problems,  and  recommended 
improvements.  These  specialists  were  not  experts  in 
software  or  human  factors  engineering,  but  they  did 
represent  a  final  authority  for  usability,  the  end  user. 

Trainees  are  also  end  users  of  the  simulation  capability. 
They  will  be  expected  to  control  the  simulation 
interface  and  based  on  resource  availability  create 
computer  models  for  sensors,  obstacles,  weapons, 
CGF’s,  and  communications.  Trainees  participated  in 
the  initial  evaluation  by  providing  baseline  training 
times  for  simulation  control  tasks  representative  of 
those  they  would  perform  during  a  simulation  exercise. 

The  complete  training  system  consists  of  several 
computers  linked  by  a  local  area  network.  However, 
for  initial  field  evaluations  only  one  laptop  computer 
was  used.  The  strategy  is  to  modify  the  control 
interface  on  the  basis  of  change  recommendations 
before  taking  the  complete  training  system  to  the  field 
for  the  final  evaluation. 

To  describe  the  usability  inspection  approach,  Brewer, 
et  al  (2002)  present  an  example  of  a  computer 
interface.  It  is  a  single  dialogue  box.  The  usability 
issue  is  whether  to  position  control  buttons  on  the 
bottom  left  or  bottom  right  of  the  dialogue  box. 
Compared  to  the  dialogue  box  described  by  Brewer,  et 
al  (2002),  the  interface  evaluated  here  is  immense.  It 
consists  of  over  450  different  controls  including 
menus,  tools,  dialogue  boxes,  intermediated  control 
windows,  and  simulation  window  symbols.  One 
recommended  approach  to  usability  testing  is  based  on 
presenting  a  control  interface  to  end  users,  not 
informing  them  how  to  use  it,  observing  if  they  can 
deduce  how  to  use  it,  and  requesting  feedback 
concerning  improvements  (Andre,  personal 
communication,  2003).  Although  this  would  be  a 
useful  approach  for  the  interface  described  by  Brewer, 


et  al.  (2002),  it  was  not  used  here.  In  addition  to  the 
great  number  of  interface  controls,  most  participants  in 
the  field  evaluation  had  no  experience  with  simulation 
capabilities  like  the  one  evaluated.  For  them  it  was  a 
novel  experience;  so  it  was  impractical  to  adopt  a 
discovery  learning  approach.  To  minimize  the 
duration  of  the  evaluation  period  while  obtaining 
meaningful  input,  it  was  necessary  to  familiarize 
participants  with  the  interface  in  advance. 

The  simulation  control  interface  used  for  the 
preliminary  evaluation  was  delivered  with  Version  2.0 
of  the  simulation  software  and  is  presented  at  Figure  2. 
It  consisted  of  a  simulation  window  in  which  a  digital 
map  is  presented;  a  menu  bar  consisting  of  menus, 
menu  options,  and  menu  option  labels;  a  tool  bar 
consisting  of  tool  bar  buttons  and  information 
windows;  a  pop-up  menu  presented  in  the  simulation 
window  and  accessed  through  a  right  mouse  click,  a 
mouse,  and  a  standard  computer  keyboard.  The  radio 
microphone  and  headset  and  simulation  logger  were 
not  included  in  the  preliminary  evaluation.  These 
interface  devices  will  be  tested  during  evaluation  of  the 
complete  training  system. 

One  of  the  greatest  obstacles  to  incorporating  such 
simulations  into  formal  training  is  the  support  costs. 
For  training  simulation  systems  currently  available,  on¬ 
site  technicians  are  required  to  design  and  develop 
simulation  exercises  in  support  of  learning  objectives 
and  must  be  present  during  the  instructional  event  to 
control  the  exercise,  serve  as  role  players,  task  CGF’s, 
and  provide  simulation  replays  to  support  after-action 
reviews.  Costs  of  support  requirements  are  a  barrier  to 
fielding  simulation  technology  for  formal  training; 
hence,  design  and  development  of  a  simulation  control 
interface  that  can  be  directly  used  by  instructors  and 
trainees  is  an  important  research  and  development 
objective.  At  the  beginning  of  the  project,  a  usability 
standard  was  established  and  defined  as  the  time 
required  to  learn  to  use  the  device.  The  expectation  is 
the  shorter  the  training  time;  the  greater  the  usability. 
The  intent  is  to  compare  the  usability  standard  with 
observed  and  maximum  training  times  as  a  guide  for 
system  modifications.  Training  time  standards  are  that 
instructors  will  be  able  to  be  trained  to  use  the  device 
in  2  hours  and  students  will  be  trained  in  30  minutes. 
Evaluation  of  training  time  was  one  of  the  objectives  of 
the  initial  evaluation. 

To  begin  the  evaluation,  each  participant  was  presented 
a  briefing  describing  the  complete  training  system  and 
the  training  concept.  After  the  briefing,  they  were 
asked  to  read  and  sign  a  disclosure  and  consent  form 
and  to  complete  a  background  questionnaire  to  obtain 
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information  about  their  time  in  service,  training,  and 
experience  with  computers.  The  evaluation  was 
conducted  one  participant  at  a  time.  This  approach 
minimized  adverse  impact  on  day-to-day  activities  of 
the  training  group  that  could  have  occurred  if  several 
participants  were  tasked  to  support  the  evaluation  in 
mass.  Each  participant  was  trained  to  use  the  control 
interface  illustrated  in  Figure  2  and  their  training  time 
was  recorded.  It  was  explained  to  each  participant  that 
they  were  not  being  evaluated;  rather  their  training 
time  was  being  recorded  to  derive  an  estimate  of 
training  time  for  evaluating  the  device.  Threshold 
training  time  was  defined  as  the  maximum  time  that 
could  be  allocated  for  a  user  to  be  trained  to  use  the 
device.  After  training  on  all  tasks,  instructors  were 
asked  to  estimate  maximum  training  time  for 
instructors  and  for  students. 

The  simulation  control  interface  included  over  450 
control  options.  We  define  control  options  as  the  total 
number  of  menu  items,  radio  buttons  and  fill-in- the- 
blank  dialogue  boxes.  The  largest  number  of  items  on 
a  menu  was  ten  and  the  deepest  nesting  of  a  menu  was 
four  levels.  Rather  than  attempting  to  train  participants 
on  all  control  options,  samples  of  tasks  were  selected 
to  represent  those  likely  to  be  performed  during  a 
simulation  exercise.  Task  selection  involved  a  trade¬ 
off  between  practicality  and  an  evaluation  of  all  control 
options.  A  balance  was  sought  between  what 
participants  might  regard  as  an  intolerably  long 
evaluation  period  and  evaluation  of  all  control  options. 
Tasks  for  both  instructors  and  trainees  consist  of  three 
categories  representing  control  of  the  simulation 
window,  placing  resources,  and  tasking  CGF’s. 
Trainee  tasks  are  a  subset  of  instructor  tasks  and 
exclude  tasks  only  instructors  would  perform  like 
creating  threat  and  neutral  CGF’s.  Instructor  and 
trainee  tasks  are  non-random  samples  from  the 
population  of  tasks.  Rather  than  randomly  sampling 
and  presenting  tasks,  they  were  carefully  selected  and 
sequenced  for  meaningfulness.  Although  the  main 
purpose  was  to  evaluate  the  training  device,  the  field 
evaluation  was  the  first  opportunity  for  instructors  to 
observe  the  capability;  so  a  secondary  goal  was  to 
make  the  experience  meaningful.  However,  the 
experience  was  time  limited.  If  more  tasks  had  been 
selected  for  the  evaluation,  total  observed  training  time 
would  have  been  greater.  However,  the  more  tasks 
selected;  the  greater  the  length  of  the  evaluation  period, 
and  the  less  likely  it  would  be  to  obtain  participants’ 
willing  cooperation  and  meaningful  input. 

Measures  of  training  time  were  obtained  separately  for 
each  participant.  The  participant  was  told  how  to 


perform  each  task,  she  or  he  was  shown  how  to  do  it, 
and  asked  to  independently  perform  the  task  with 
assistance.  They  were  asked  to  indicate  when  they  had 
learned  to  use  interface  controls  for  the  task.  When  the 
participant  stated  she  or  he  had  learned  to  perform  the 
task,  training  time  was  declared  complete. 
Immediately  after  training  for  each  task,  the  participant 
was  asked  to  perform  the  task  independently.  It  was 
noted  whether  it  was  performed  with  our  without 
assistance.  It  was  assumed  that  if  the  participant 
performed  the  task  without  assistance,  they  had  learned 
to  use  simulation  interface  controls.  If  the  participant 
asked  for  assistance,  recorded  training  time  for  that 
task  for  that  participant  was  doubled.  Task  training 
time  was  cumulated  over  all  tasks  to  obtain  an  estimate 
of  training  time  for  a  single  participant.  Training  time 
was  averaged  over  trainees  and  instructors  separately 
to  estimate  total  training  time  for  each  group. 

Trainee  participation  was  limited  to  learning  how  to 
perform  each  task.  Trainees  did  not  provide 
evaluations  of  the  training  device.  Evaluations  were 
conducted  only  by  instructors.  After  training  for  each 
task,  the  instructor  rated  the  controls  used  to  perform 
the  task  on  clarity,  effectiveness,  efficiency,  and 
simplicity;  always  in  that  order.  Clarity  was  defined  as 
the  degree  to  which  interface  controls  were  clear  and 
understandable.  Effectiveness  was  defined  as  the 
degree  to  which  interface  controls  allowed  the  task  to 
be  performed.  This  criterion  factor  provided  an 
opportunity  for  instructors  to  recommend  additional 
functionality  to  improve  the  effectiveness  of  controls. 
Efficiency  was  defined  as  the  degree  to  which  the 
controls  used  in  performing  the  task  allowed  quick 
performance.  Simplicity  was  defined  as  the  degree  to 
which  the  logic  of  using  the  controls  was  complex  or 
easy  to  understand.  Extremes  of  the  rating  scale  for 
simplicity  were  anchored  with  verbal  anchors, 
“Extremely  High  Simplicity”  and  “Extremely  High 
Complexity”.  Rating  scales  for  clarity,  effectiveness, 
and  efficiency  used  similar  scales.  The  process  of 
obtaining  ratings  guided  the  instructor  to  think  about 
specific  criteria  for  usability  and  provided  indicators  of 
order  relationships  among  tasks  for  each  criterion. 
After  rating  the  control  interface  for  a  task  on  one  of 
the  criteria,  instructors  were  asked  to  identify  problems 
and  recommend  improvements. 

At  the  conclusion  of  the  evaluation  period,  instructors 
were  asked  if  they  agreed  that  observed  simulation 
scenarios  would  support  learning  objectives  for 
mission  planning  and  execution  of  the  defense  and 
whether  or  not  they  believed  the  simulation  capability 
would  add  value  to  training. 
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RESULTS 

A  total  of  20  instructors  participated  in  the  evaluation 
(10  instructors  from  the  96th  Security  Forces  Ground 
Combat  Training  Squadron  located  at  Eglin  AFB,  FL 
and  10  instructors  from  the  37th  Training  Group  located 
at  Lackland  AFB,  TX).  The  first  evaluation  was 
conducted  at  the  96th  Ground  Combat  Training 
Squadron  where  the  original  task  list  consisted  of  24 
tasks.  Each  of  the  first  7  instructors  dedicated  6  hours 
to  the  evaluation  including  lunch  and  breaks.  After 
instructor  7,  the  task  list  was  reduced  to  18  tasks  to 
minimize  the  burden  on  participants.  The  18  task  list 
required  approximately  4  hours  for  each  instructor. 
The  18  task  list  was  used  for  all  instructors  from  the 
37th  Training  Group.  The  results  reported  in  this  paper 
are  for  the  17  tasks  common  to  the  first  and  second 
task  list.  Participants  also  included  10  trainees  from 
the  343  Training  Squadron  who  had  recently  graduated 
from  security  forces  initial-skills  training  and  were 
awaiting  assignments. 

On  average,  the  instructors  were  29.35  years  of  age. 
There  were  3  captains,  1  senior  master  sergeant,  1 
master  sergeant,  4  technical  sergeants,  9  staff 
sergeants,  and  2  senior  airmen.  All  enlisted  personnel 
serving  as  instructors  were  at  the  journeyman  skill 
level  or  higher  within  the  security  forces  career  field. 
They  had  an  average  of  9.4  years  of  service  and  1.7 
years  in  their  current  position.  They  indicated  they 
spend  an  average  of  38.6  hours  per  week  using 
computers  and  in  the  preceding  year  played  computer 
games  an  average  of  5.65  times.  Trainees  participated 
in  learning  how  to  perform  13  tasks. 


•  Pan,  zoom  in/out  on  map 

•  Use  range  and  bearing  tool  to  determine  distance 
on  map 

•  Determine  height  using  contour  lines  and  display 

•  Place  CGF  in  fighting  position,  set  fields  of  fire 

•  Determine  dead  space  using  intervisibility  tool 

•  Assign  field  phones  to  fighting  positions 

•  Create,  place,  move,  orient,  delete 

•  Obstacle 

•  Sensor 

•  CGF 

•  Primary  Fighting  Position 

•  Assign  radios  and  call  signs  to  CGF’ s  and  generate 
location  reports 

•  Task  vehicle  to  conduct  patrol  along  assigned 
route 

•  Retask  CGF  to  go  to  map  location 

Instructors  were  tested  on  four  additional  tasks. 

•  Create,  place,  move,  orient,  delete 

•  Threat  CGF’ s 

•  Neutral  CGF’ s 

•  Facilities  and  Aircraft 

•  Create  instructor-generated  SITREP 

On  average,  trainees  were  19  years  of  age.  Five  of  the 
trainees  possessed  the  rank  of  airman  first  class  and  5 
possessed  the  rank  of  airman  basic.  All  trainees  had 
acquired  the  apprentice  skill  level.  They  had  an 
average  of  6  months  of  service.  They  indicated  they 
spend  an  average  of  4  hours  per  week  using  the 
computer  and  in  the  preceding  year  played  computer 
games  an  average  of  3  times. 


Figure  3.  Average  task  training  time  for  20  instructors 
(Summation  over  17  tasks  =  1  hr  16 


1  2  3  4  5  6  7  8  9  10  1 1  12  13  14  15  16  17 

Task  number 

Maximum  allowable  instructor  training  time  =  4  hr  39 
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Figure  3  presents  the  observed  task  training  time  for  17 
tasks  averaged  over  instructors.  Most  tasks  required  5 
minutes  or  less  training  time.  The  instructor  training 
time  usability  standard  was  2  hours.  Figure  4  indicates 
that  the  observed,  average  training  time  for  instructors 
summed  over  the  17  task  sample  is  1  hour  and  16 
minutes.  Observed  training  time  for  the  17  task  sample 
fell  below  the  pre-established  two-hour  standard.. 
After  task  training,  each  instructor  was  asked  to 
estimate  the  maximum  amount  of  time  that  could  be 
allocated  for  instructors  to  be  trained  to  use  the  device. 


Maximum  instructor  training  time  was  estimated  to  be 
4  hours  and  39  minutes,  on  average. 

Figure  4  presents  the  observed  training  time  for  13 
tasks  averaged  over  trainees.  Most  of  the  tasks 
required  5  minutes  or  less  training  time.  For  trainees, 
the  training  time  usability  standard  is  30  minutes. 
Figure  4  indicates  that  the  observed,  average  training 
time  for  students  summed  over  the  13  task  sample  is  56 
minutes.  Observed  training  time  for  the  13  task  sample 
exceeded  the  pre-established  30  minute  standard  for 


Figure  4.  Average  task  training  time  for  1 0  trainees 
(Summation  over  13  tasks  =  56  min  2 


T  ask 

Maximum  allowable  student  training  time  =  4  hr  9 


student  training  time.  Each  instructor  was  asked  to 
estimate  the  maximum  amount  of  time  that  could  be 
allocated  for  students  to  learn  to  use  the  device. 
Instructors  estimated  the  maximum  student  training 
time  at  4  hours  and  9  minutes,  on  average.  For 
instructors,  the  procedure  was  to  learn  to  use  interface 
controls  then  rate  the  control  interface  on  clarity, 
effectiveness,  efficiency,  and  simplicity;  always  in  this 
order.  Figure  5  presents  average  interface  usability 
ratings  for  each  criterion  factor.  Even  though  data 
collection  was  arranged  on  a  task-by-task  basis, 
instructors  were  told  that  the  collection  of  interface 
controls  used  to  perform  the  task  were  the  target  for 
each  rating.  For  each  usability  factor,  a  rating  value  of 
5  represents  an  “Average”  rating,  while  7  and  8 
represent  “High”  and  “Very  High”.  Aggregate  ratings 
for  all  tasks  and  criterion  factors  were  above  average. 
These  results  indicate  instructors  believed  the  control 
interface  was  easy  to  understand,  it  was  effective,  it 


allowed  them  to  accomplish  tasks  quickly,  and  the 
logic  of  operations  required  for  using  the  controls  was 
easy  to  understand. 

Although  the  instructors  were  enthusiastic  about  the 
ease  of  use  and  training  potential  of  the  system, 
everyone  always  has  recommendations  for 
improvements.  The  20  instructors  provided  a  total  of 
310  change  recommendations.  These  change 
recommendations  consisted  of  suggestions  to  improve 
deficiencies  in  usability  and  suggestions  to  add 
functionality.  The  most  frequently  recommended 
changes  are  listed  below: 

•  Migrate  as  many  options  as  possible  from  the  top 
of  display  menu  to  the  right  click  menu. 

•  Make  map  panning  and  zooming  functions  more 
efficient 
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Fix  grid  lines  at  1  Km  instead  of  making  them 
dependent  on  zoom  level 
Simplify  turning  contour  lines  on  and  off 
Show  max  range  of  weapons  in  addition  to  max 
effective  range  currently  shown 


Include  display  of  LOGDET  assets  used 
Implement  copy/paste  option  for  placing  assets 
Allow  instructor  to  cause  communications  failures 
Simplify  tool  for  generating  friendly  SITREPS 
Modify  colors  of  asset  icons 


Figure  5.  Average  interface  usability 


Finegold,  L.S.,  Withman,  R.L.,  &  Kuperman,  G.G. 
(2001)  found  similar  results  concerning  map  zooming 
and  panning.  They  recommended  that  users  be  able  to 
zoom  directly  to  the  desired  magnification  level  and 
map  center  without  going  through  intermediate  zoom 
values  and  pans.  These  changes  along  with  the  great 
majority  of  other  changes  have  been  implemented  in 
the  next  spiral  development,  but  instructor  evaluation 
will  occur  after  the  due  date  of  this  publication. 

After  task  training  and  the  process  of  providing 
usability  ratings,  each  instructor  was  asked  to  provide 
summary  evaluations.  All  20  instructors  except  one 
agreed  the  device  would  add  value  to  training.  All 
agreed  the  draft  training  scenarios  they  observed  would 
support  learning  objectives  for  security  planning;  and 
all  but  one  instructor  agreed  the  capability  would 
support  learning  objectives  for  execution  of  the 
defense. 

DISCUSSION 

Although  observed  training  times  provide  useful 
baseline  estimates  of  usability,  they  have  limitations. 
The  initial  evaluation  did  not  include  the  simulation 


logger  or  the  radio  microphone  or  headset.  If  tasks  for 
these  devices  had  been  included,  training  time  would 
have  been  greater.  Training  time  for  these  devices  will 
be  estimated  when  the  complete  training  system  is 
evaluated. 

Because  it  was  necessary  for  reasons  of  practicality  to 
limit  the  number  of  tasks  evaluated,  training  times  are 
underestimates.  Only  a  representative  subset  of  tasks 
were  used  for  instructors  and  trainees.  Observed 
training  time  underestimates  total  training  time  for  all 
450  interface  control  options. 

For  instructors,  total  observed  training  time  is  an 
overestimate.  During  the  training  period  for  each  task, 
instructors  often  discussed  problems  and  recommended 
needed  capabilities.  The  result  was  that  total  observed 
training  time  includes  time  required  for  training  and 
time  for  non-training  interactions. 

Better  estimates  of  observed  training  time  are  needed. 
It  would  be  desirable  to  obtain  a  more  accurate 
estimate  of  observed  training  time  that  excludes  time 
for  non-training  interactions,  includes  tasks  for  use  of 
the  radio  headset,  and  simulation  logger,  and  a  greater 


Interservice/Industry  Training,  Simulation,  and  Education  Conference  (I/ITSEC)  2003 


number  of  representative  tasks  while  maintaining  a 
reasonable  time  for  the  evaluation  period.  Multiple 
evaluation  periods  may  be  necessary  if  acceptable  to 
participants. 

Even  though  there  are  limitations  in  observed  training 
time,  it  is  important  to  understand  the  alternatives.  If 
observed  training  time  for  instructors  were  doubled,  it 
would  still  be  less  than  the  allowable  training  time 
specified  by  the  instructors  and  substantially  less  time 
than  required  for  other  training  simulation  capabilities 
available  today.  For  example,  consider  a  joint  tactical 
simulation  currently  in  use.  On  a  separate  occasion, 
the  co-author  observed  security  forces  personnel  being 
trained  on  this  system  to  support  installation  security 
evaluations.  Observation  of  “user  training”  for  the 
system  indicated  7  security  forces  personnel  varying  in 
rank  from  staff  sergeant  to  master  sergeant 
satisfactorily  learned  to  task  CGF’s  to  move, 
dismounted  and  mounted,  and  shoot,  direct  and 
indirect- fire  weapons,  in  1  and  Vi  eight-hour,  training 
days.  Experts  indicated  they  had  learned  to  operate  the 
system  over  a  period  of  months.  If  the  training  time 
for  the  system  is  the  comparison  point,  SecForDMT 
offers  a  significant  advantage  by  avoiding  lengthy 
training  times  for  end  users.  This  reduced  training 
time  can  be  attributed  primarily  to  its  greater  usability. 

LESSONS  LEARNED 

Even  for  a  simulation  with  450  functions  the  use  of 
established  usability  techniques  can  make  the  system 
usable  by  novices  in  a  very  short  time.  The  use  of  a 
mixed  initiative  dialogue  with  direct  manipulation  of 
data  has  been  shown  to  be  an  effective  means  of 
allowing  novices  to  operate  a  war  game  simulation 
quickly.  Control  of  CGF’s  and  other  assets  via  right- 
click  menu  selections  was  intuitive  and  easy  for 
novices  to  master.  Since  panning  and  zooming  a 
display  can  consume  significant  time,  providing 
trainees  with  tools  to  pan  and  zoom  and  center  a 
display  in  one  step  without  going  through  intermediate 
zooms  is  critical.  To  take  advantage  of  population 
stereotypes,  design  the  interface  similar  to  Windows 
and  video  game  user  interfaces.  Although  students’ 
observed  training  time  of  56  minutes  exceeded  the  self- 
imposed  30-minute  standard,  the  maximum  training 
time  available  for  training  students  (4  hours)  suggests 


that  the  capability  may  still  be  useful  for  training.  Due 
to  the  emphasis  on  usability  in  the  design  of 
SecForDMT,  instructors  rated  control  interface 
usability  high  and  very  high  for  almost  all  tasks.  Most 
importantly,  our  research  indicates  that  simulation 
interfaces  can  be  designed  for  control  by  trainees  and 
instructors  without  an  intermediate  controller. 
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